Skip to content

Enhance error logs for invalid custom fields #235

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Nov 4, 2025
Merged

Enhance error logs for invalid custom fields #235

merged 5 commits into from
Nov 4, 2025

Conversation

@kaisecheng
Copy link
Contributor

@kaisecheng kaisecheng commented Oct 30, 2025 •
edited
Loading

add error logs when MaxMind databases fail to deserialize custom field

Fix: #233

How to test this PR

The instruction is copied from #226

  1. modify a sample test DB hosted in this project, updating a record with a custom information is_in_european_union
go build
./mmdb-from-go-blogpost
  1. run the following pipeline, setting the path to the DB file
input {
 generator {
   message => '{"ip": "216.160.83.58"}'
   codec => json
   count => 1
 }
}


filter {
 geoip {
   ecs_compatibility => disabled
   database => "/your/path/to/GeoIP2-Country-Test.mmdb"
   source => "ip"
   target => "[client][ip]"
 }
}

output {
 stdout{}
}
  1. Logstash should show error logs
[2025-10-30T15:54:30,874][ERROR][org.logstash.filters.geoip.GeoIPFilter][main][2e27f9fca6263b0b077b091c2ad8f92a639707d3727f9184c31c3ffa4de4b004] Error from GeoIP database. field=[ip], value=216.160.83.58
org.logstash.filters.geoip.GeoIPFilter$GeoIp2InvalidCustomFieldException: The database contains invalid custom field, which caused deserialization to fail.
	at org.logstash.filters.geoip.GeoIPFilter.retrieveCountryGeoData(GeoIPFilter.java:360) ~[logstash-filter-geoip-7.3.1.jar:?]
	at org.logstash.filters.geoip.GeoIPFilter.handleEvent(GeoIPFilter.java:179) ~[logstash-filter-geoip-7.3.1.jar:?]
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
	at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:285) ~[jruby.jar:?]
	at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:152) ~[jruby.jar:?]
	at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:24) ~[jruby.jar:?]
	at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:86) ~[jruby.jar:?]
	at org.jruby.ir.targets.indy.InvokeSite.performIndirectCall(InvokeSite.java:735) ~[jruby.jar:?]
	at org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:657) ~[jruby.jar:?]
	...
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111) ~[jruby.jar:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "parameters[index]" is null
	at com.maxmind.db.Decoder.decodeMapIntoObject(Decoder.java:446) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decodeMap(Decoder.java:342) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decodeByType(Decoder.java:162) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decode(Decoder.java:151) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decodeMapIntoObject(Decoder.java:429) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decodeMap(Decoder.java:342) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decodeByType(Decoder.java:162) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decode(Decoder.java:151) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Decoder.decode(Decoder.java:76) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Reader.resolveDataPointer(Reader.java:275) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.db.Reader.getRecord(Reader.java:185) ~[maxmind-db-2.1.0.jar:?]
	at com.maxmind.geoip2.DatabaseReader.get(DatabaseReader.java:263) ~[geoip2-2.17.0.jar:2.17.0]
	at com.maxmind.geoip2.DatabaseReader.getCountry(DatabaseReader.java:309) ~[geoip2-2.17.0.jar:2.17.0]
	at com.maxmind.geoip2.DatabaseReader.country(DatabaseReader.java:292) ~[geoip2-2.17.0.jar:2.17.0]
	at org.logstash.filters.geoip.GeoIPFilter.retrieveCountryGeoData(GeoIPFilter.java:358) ~[logstash-filter-geoip-7.3.1.jar:?]
	... 67 more
[2025-10-30T15:54:30,880][DEBUG][logstash.filters.geoip   ][main][2e27f9fca6263b0b077b091c2ad8f92a639707d3727f9184c31c3ffa4de4b004] IP 216.160.83.58 was not found in the database {:event=>#<LogStash::Event:0x69c6e28c>}
{
          "tags" => [
        [0] "_geoip_lookup_failure"
    ],
          "host" => {
        "name" => "your-mac-pro"
    },
    "@timestamp" => 2025-10-30T15:54:30.783252Z,
         "event" => {
        "original" => "{\"ip\": \"216.160.83.58\"}",
        "sequence" => 0
    },
      "@version" => "1",
        "source" => {},
            "ip" => "216.160.83.58"
}

@kaisecheng kaisecheng force-pushed the add_logs_invalid_custom_field branch from 1d48c77 to e4a5c39 Compare October 30, 2025 17:20
@kaisecheng kaisecheng requested a review from alexcams November 4, 2025 10:42
alexcams
alexcams reviewed Nov 4, 2025
View reviewed changes
src/main/java/org/logstash/filters/geoip/GeoIPFilter.java Outdated
static class GeoIp2InvalidCustomFieldException extends GeoIp2Exception {
public GeoIp2InvalidCustomFieldException(Throwable cause) {
super("invalid custom field", cause);
super("The database contains invalid custom field, which caused deserialization to fail.", cause);
Copy link

@alexcams alexcams Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: There is double tab.

alexcams
alexcams approved these changes Nov 4, 2025
View reviewed changes
Copy link

@alexcams alexcams left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! with minor to remove double tab.

@kaisecheng kaisecheng merged commit d048685 into logstash-plugins:main Nov 4, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alexcams alexcams alexcams approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Error logging is insufficient to track down cause of error

2 participants

@kaisecheng @alexcams